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The Relationship of a Systemic Student Support Intervention to 
Academic Achievement in Urban Catholic Schools 

Katherine A. Shields, Mary E. Walsh, and Terrence J. Lee-St. John 
Boston College 


Much of the achievement gap between rich and poor students can be attributed to 
out-of-schoolfactors, yet few schools have a comprehensive, coordinated system for 
addressing students’ nonacademic needs. Within a group of Catholic schools located 
in one city, this study examined academic achievement on the Stanford Achieve¬ 
ment Test battery in mathematics, reading, and language among second- through 
eighth-grade students participating in such an intervention, and compared the re¬ 
sults with those of similar nonparticipating students in nearby cities. Using hierar¬ 
chical longitudinal growth modeling and adjustingfor demographic characteristics, 
this study found that students in intervention schools outperformed the comparison 
group on average in sixth-grade mathematics. Intervention students also expe¬ 
rienced significantly higher rates of growth in achievement than the comparison 
group in all three subjects. The results suggest that systemic service provision models 
have the potential to help urban Catholic schools meet their mission ofeducating the 
whole child and serving the poorestfamilies. 

Keywords 

Catholic education, longitudinal growth analysis, achievement gap 

C atholic schools have a long tradition of serving families living in pov¬ 
erty in the nation’s cities. Historically, the parochial school sought to 
meet the needs of its surrounding parish community (Bryk, Lee, 6c 
Holland, 1993). In recent decades, as parishioners have dispersed from urban 
areas, leaving lower-income and increasingly non-Catholic student bodies in 
urban schools (Goldschmidt 6c Walsh, 2012), this commitment has broadened 
to encompass service to urban communities of various faiths. Providing a 
quality education to disadvantaged urban families aligns with the church’s so¬ 
cial teachings and mandate to serve the poor (Grace 6c O’Keefe, 2007). The 
recent National Standards and Benchmarks for Effective Catholic Elementary 
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and Secondary Schools (Ozar 8c Weitzel-O’Neill, 2012) reinforce this com¬ 
mitment, calling for schools to provide “programs and services aligned with 
the mission to enrich the academic program and support the development 
of student and family life” (p. 12). However, private and public schools alike 
struggle with how best to support families to counteract the myriad ill effects 
of poverty, particularly those that singly and in combination hinder academ¬ 
ic progress. This study examines a whole-child strategy for serving students’ 
nonacademic needs in urban Catholic schools, and the academic outcomes of 
participating students. 

Out-of-School Factors and the Achievement Gap 

Poverty affects children’s development and academic performance through 
a range of mechanisms. Exposure to parental stress in utero and during early 
childhood can affect neurological development (Shonkoff et al., 2012). Re¬ 
duced financial resources and associated stressors affecting lower-income par¬ 
ents may limit their capacity to create a psychosocially rich environment for 
their children through responsiveness and warmth (Dearing 8c Taylor, 2007), 
participation in enriching after-school activities (Dearing et al., 2009), expo¬ 
sure to a wide vocabulary (Hart 8c Risley, 1995), investment in a cognitively 
rich environment, and parenting practices that support development (Yeung, 
Linver, 8c Brooks-Gunn, 2002; Rothstein, 2004; Sirin, 2005). Placed at risk 
by societal structures, low-income and historically disadvantaged minority 
children are more likely than nonpoor and White peers to have poor health 
status, starting with lower birth weight; greater food insecurity and malnutri¬ 
tion; greater exposure to environmental hazards that impede cognitive de¬ 
velopment; and more frequent disruptive moves between schools (Barton 8c 
Coley, 2009). 

These factors outside school may account for as much as two-thirds of 
the variance in academic achievement, a pattern that emerged in the his¬ 
toric Coleman Report in 1966 (Coleman et al., 1966) and was replicated in 
more recent studies (Phillips, Brooks-Gunn, Duncan, Klebanov, 8c Crane, 
1998; Rothstein, 2010). The achievement gap related to income has grown as 
the divide between the income levels of rich and poor families has widened 
(Duncan 8cMurnane, 2011; Reardon, 2011). Furthermore, the extent of the 
gap in the prevalence of risk factors between lower-income and minority 
children and their peers has largely remained unchanged over recent years 
(Barton 8c Coley, 2009). Researchers have thus argued that the achievement 
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gap at scale cannot be closed without addressing poverty and its effects (Ber¬ 
liner, 2006, 2013; Rothstein, 2004). 

The Context of Urban Catholic Schools 

Urban families make up close to half of Catholic schools’ constituencies. 
In 2012-2013,41% of Catholic schools nationwide were located in urban areas; 
in New England, 15% of elementary schools were further classified as inner- 
city. Among New England Catholic elementary and middle school students, 
8% was Black, 5.1% Asian, and 4.7% multiracial; 8.2% was Hispanic/Latino. 

As of 2012-2013, the Archdiocese of Boston, the location of the present study, 
served more than 41,000 students at more than 100 schools in the elemen¬ 
tary, middle, and secondary levels across Eastern Massachusetts, including 
17 schools in the city of Boston (McDonald &c Schultz, 2013). The Boston 
Archdiocese reported that across all of its schools, urban and nonurban, 5.1% 
of students received Title I services targeting low-income families in 2012- 
2013, and 8.2% received federally subsidized lunches; the schools within the 
city of Boston itself, however, served substantially higher numbers of families 
in poverty than the average for the Archdiocese. 

Urban Catholic schools like those in the Boston area serve relatively high 
percentages of low-income and historically disadvantaged minority students, 
facing many of the same barriers as their neighbors attending public schools 
(Bryk et al., 1993; O’Keefe & Scheopner, 2009). In a 2003 survey of Catholic 
school staff in a northeastern urban diocese, 72% of teachers and 86% of prin¬ 
cipals saw nonacademic issues as a barrier to their students’ learning (Walsh 
Sc Goldschmidt, 2004). This perception is borne out by Catholic school 
students’ scores on the National Assessment of Educational Progress, the 
“Nation’s Report Card,” on which lower-income students eligible for free or 
reduced-price lunch perform substantially lower on average than their mid¬ 
dle- and upper-income noneligible peers. 1 Catholic schools are particularly 
well-suited to offer comprehensive support for out-of-school issues, given 
their stated mission to educate the whole child and the parish’s historical role 
as a community service provider (Walsh & Goldschmidt, 2004). 


1 NAEP Data Explorer, fourth-grade mathematics and reading, 2011 and 2013, accessed 
at http://nces.ed.gov/nationsreportcard/naepdata. 
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Interventions to Address Out-of-School Factors 

Although most public and private K-12 schools offer some form of sup¬ 
port for students’ nonacademic needs, these services are often fragmented 
and uncoordinated, lacking systemic organization (Lean 8c Colucci, 2010). 
School counselors, working both within the school bureaucracy and in rela¬ 
tionship to external community agencies, are hampered in their work by this 
lack of structure (Bridgeland 8c Bruce, 2011). To address these institutional 
issues, best practice guidelines recommend the development of systematic, 
comprehensive approaches to meeting students’needs (Adelman 8c Taylor, 
2006; Marx, Wooley, 8c Northrop, 1998). 

Children who receive appropriate, sustained, well-targeted supports face 
fewer outside-school stressors and develop greater self-regulation of social- 
emotional and cognitive functions (Shonkoff 8c Phillips, 2000), thereby 
allowing students to arrive at school more ready to learn (Ayoub 8c Fischer, 
2006; Noguera, 2011). Research has demonstrated links between improve¬ 
ments in certain noncognitive factors and positive academic achievement 
outcomes (Duckworth 8c Seligman, 2005; Heckman, 2008; Heckman, Pinto, 
8c Savelyev, 2012). Various interventions that address those factors directly 
have demonstrated academic gains (Dignath 8cBiittner, 2008; Lassen, Steele, 
8c Sailor, 2006; Villares, Frain, Brigman, Webb, 8c Peluso, 2012). For ex¬ 
ample, increased family involvement in the child’s education has been linked 
with reductions in the literacy achievement gap (Dearing, Kreider, Simpkins, 
8c Weiss, 2006). 


The City Connects Intervention Model 

City Connects is an intervention that implements theoretically guided 
practices for student support in high-poverty, urban schools. Its design is 
aligned with the principles of best practice in student support (Adelman 8c 
Taylor, 2006; Marx et al., 1998; Walsh, 8c Brabeck, 2006). Developed in 2001 
through a collaboration between Boston College, the Boston Public Schools, 
and area community agencies, the intervention was implemented in 64 public 
and Catholic schools across three states in the 2014-2015 school year. City 
Connects began formal implementation in Archdiocese of Boston elemen¬ 
tary and K-8 schools in 2008-2009. In the 2014-2015 school year, 17 Catholic 
elementary schools in the City of Boston were implementing the model. 

City Connects’s approach is comprehensive, addressing the academic, 
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social-emotional, health, and family strengths and needs of each and every 
student in a school. A master’s-trained, licensed school counselor or school 
social worker in each school, the City Connects school site coordinator , is at 
the core of the intervention. Figure 1 illustrates the City Connects theory of 
change. 



• Collaborates with each 
teacher to develop a 
uniquely tailored student 
support plan for every 
child 

• Leverages both school- 
and community-based 
services and enrichment 
opportunities 

• Partners with families to 
ensure service delivery 

• Systematically uses data 
to improve practice and 
assess outcomes 


• Students' strengths and 
needs are addressed 

• Teachers shape 
instruction to meet 
student challenges 

• Principals have more time 
to focus on instruction 

• Families have a trusted 
person to facilitate access 
to supports 

• Community Providers can 

better tailor and deliver 
services 


Figure 1. City Connects’s theory of change. 


• Significant improvements 
in student achievement 
and ability to thrive 

• Thriving: Behavior, Work 
Habits, and Effort 

• Academic Performance: 
Elementary School 
Grades 


Lasting Posi ive Academic 
Effects 

Improved Life Chances 


Each classroom teacher and the school site coordinator meet to discuss 
every student in the class using standardized guiding questions. Student 
strengths and needs across four developmental domains—academic, socio- 
emotional, health, and family—are electronically documented, and students 
are grouped into three tiers, with Tier 1 representing strengths and minimal 
risks, Tier 2 representing strengths and mild to moderate risks, and Tier 
3 representing strengths and severe risks. The school site coordinator and 
teacher collaborate to identify types of services and supports that could 
promote student strengths and meet needs. Using a computer-based tool 
designed for the intervention, site coordinators then identify specific service 
providers based on factors such as service type(s), ages served, location, trans¬ 
portation requirements, and family capacity to support participation (e.g., ac- 
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cess to insurance). Site coordinators connect children and their families with 
service providers, monitor service quality and fit, and maintain partnerships 
with community service providers. 

For children identified by teachers, coordinators, or other school staff as 
having intensive needs, coordinators arrange individual student reviews with 
a wider team of professionals—school psychologists, teachers, principals, 
nurses, and community agency staff—to develop specific goals and strategies 
for the student. Examples of commonly provided services include before- 
and after-school programs, sports, mentoring, tutoring, social skills interven¬ 
tions, health screenings, family counseling, and food or clothing donations. 
These services are sometimes available within the school, but are also pro¬ 
vided by community agencies. 

The coordinator uses a proprietary web-based system to track the service 
plan for each student. A documented, standardized set of practices, oversight 
mechanisms, and fidelity tools guide implementation across sites and ser¬ 
vices. School site coordinators participate in rigorous training in the model 
and continuous professional development, including biweekly meetings and 
webinars. 

Guiding the City Connects approach is continuous formative and sum- 
mative evaluation of student outcomes, including rigorous, longitudinal 
evaluation of academic achievement. In the Boston Public Schools (BPS), 
evaluation has included tracking student performance on statewide stan¬ 
dardized assessments; students in City Connects BPS schools were found to 
achieve higher report card scores in elementary school and higher grades and 
standardized test scores in middle schools than comparison students (Walsh 
et al., 2014). Although private schools in Massachusetts are not required to 
participate in the statewide assessment, schools in the Archdiocese of Boston 
administer the Stanford Achievement Test, a nationally normed assessment 
battery. These test results offer a valid, reliable way to analyze the association 
between City Connects participation and changes in academic achievement 
over time. 


Research Questions 

Using a student-level dataset comprising achievement test scores and de¬ 
mographic data from Catholic schools participating in City Connects as well 
as nonparticipating Catholic comparison schools, this study contributes to 
our understanding of how systemic student support interventions can address 
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the achievement gap in the context of urban Catholic schools. The analysis 
addressed the following research questions: 

• Does academic achievement in mathematics, reading, and language differ 
between students who participate in a systematic support intervention and 
those attending comparison schools, after controlling for demographic 
characteristics? 

• Do the rates of growth in achievement differ between intervention and 
comparison students? 

• Among intervention students, does the number of years of exposure to the 
City Connects program have an association with their rates of growth in 
achievement? 

These questions were considered in the context of four student char¬ 
acteristics that have been linked to academic achievement in the research 
literature. Gender is a perennial subject of study, particularly in relation to 
mathematics achievement (Kenney-Benson, Pomerantz, Ryan, 8c Patrick, 
2006; Robinson 8c Lubienski, 2011; Spencer, Steele, 8c Quinn, 1999), although 
some recent studies have suggested that girls and boys are now performing at 
similar levels (Hyde, Lindberg, Linn, Ellis, 8c Williams, 2008). As discussed 
abo vz, family income level has strong links to children’s performance in school 
(Berliner, 2006; Yeung et al., 2002). Race is strongly associated with educa¬ 
tional outcomes, even after accounting for income, and exhibits interactions 
with measures of income and wealth (Elliott, Jung, Kim, 8c Chowa, 2010; 
Riegle-Crumb 8c Grodsky, 2010; Zhan, 2006;). Finally, children receiving 
special education services experience a particular set of educational interven¬ 
tions whose effects on academic achievement vary with the type of disability 
identified, the quality of interventions, how early the intervention begins, and 
other factors (Ehrhardt, Huntington, Molino, 8c Barbaresi, 2013; Reynolds 8c 
Wolf, 1999). 

Methodology 


Data Sources 

The analyses reported below were based on a longitudinal student-level 
dataset comprising 3,628 students from 17 Boston Catholic elementary/ 
middle schools participating in the intervention, and 3,323 students from 
10 comparison schools in the Boston Archdiocese. Because all Archdiocese 
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schools in the City of Boston participated in City Connects, a sample of 
10 nontreatment comparison schools was drawn from more than 60 Arch¬ 
diocese schools serving grades K to 8 located outside Boston. Schools were 
selected that had similar geographic characteristics (midsize urban centers in 
surrounding communities) and demographic profiles (percentages of racial/ 
ethnic minority students and lower-income families higher than the state 
average) to treatment schools. 

Data were analyzed for four school years, from 2009-2010 through 
2012-2013, for grades two through eight. Data were collected from multiple 
sources. Intervention schools entered demographic data about students in 
a proprietary Web-based project database. In addition, all intervention and 
comparison schools provided City Connects with class lists, including de¬ 
mographic data, annually. The Catholic Schools Office of the Boston Arch¬ 
diocese provided student Stanford Achievement Test scores and served as an 
additional source of demographic data. Finally, the Catholic Schools Foun¬ 
dation provided family income data for a subset of students who applied for 
scholarship aid. 

Measures 

The Stanford Achievement Test, 10th edition (Stanford 10) is a standard¬ 
ized, norm-referenced assessment of student academic achievement (Pearson, 
2004). It offers a battery of multiple-choice tests in mathematics, language, 
reading, science, and social science for kindergarten through grade 12. Con¬ 
tent is linked to state and national standards in each area. The reading assess¬ 
ment includes decoding, vocabulary, and comprehension; the language assess¬ 
ment covers word- and sentence-level skills, mechanics, and expression; and 
the mathematics assessment tests skills in both computational proficiency 
and problem-solving. Because the tests are vertically scaled across grades, a 
student’s scale scores can be compared from one grade to another to measure 
growth in achievement over time. The Stanford 10 was administered each 
spring during the four years of the study. Stanford 10 mathematics, reading, 
and language scale scores were analyzed as outcome variables in this study. 

Three variables were used to represent participation in the interven¬ 
tion: a dummy variable indicating whether or not the student attended a City 
Connects in the Catholic Schools (CCCS) intervention school in any year in 
the study period; a cumulative dosage variable for each year, representing the 
number of years to date within the study period that the student had attend- 
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ed intervention schools; and a maximum dosage variable representing the 
total number of years the student was enrolled in intervention schools during 
the study period. The 3% of students who switched between intervention and 
comparison schools during the study period were excluded from analysis; the 
1% of students who repeated a grade was retained in the sample. 

Student demographic covariates related to achievement were employed as 
controls. Gender and race (with categories Asian, White, Black, Hispanic, 
and other/more than one race) were represented in the model. In addition, 
the analysis included a control variable indicating whether or not the stu¬ 
dent had an Individualized Education Plan (IEP) in any year of the study, 
and a control variable designating eligibility for free or reduced-price lunch 
through the National School Lunch Program in any year of the study. 


Missing Data 

Gender, race/ethnicity, and IEP status were missing for less than 1% of 
students, while free/reduced-price lunch eligibility was missing for 18%. Stu¬ 
dents were excluded from analysis if their records were missing one or more 
of these four control variables. The resulting analytic sample contained 3,216 
intervention school students and 2,403 comparison school students. With 
a total of 19% of students excluded from the sample due to missing demo¬ 
graphic information, any systematic factors related to the absence of these 
data could bias results. It should also be noted that a higher percentage of 
comparison students (28%) than intervention students (11%) were excluded 
on this basis, likely due to more complete record-keeping available through 
the project database. An alternative set of analyses was run in which cases 
with missing demographic data were retained and flagged. The direction and 
magnitude of the treatment coefficients (indicating whether or not students 
participated in CCNX, and for how many years) were similar to coefficients 
estimated with the original sample, suggesting that results were not sensitive 
to the choice of method (see further discussion under results). 

Analytic Method 

To answer the first research question ( Does academic achievement differ 
between intervention and comparison students?), cross-sectional differences 
between the two groups were estimated for the sixth grade. Although data 
were included in the analysis for the seventh and eighth grades, sixth grade 
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was chosen as the endpoint for the cross-sectional analysis because fewer 
Catholic schools offered upper grades, yielding a relatively small number 
of student records. To address the second research question (Do the rates of 
growth in achievement differ between intervention and comparison students?), 
the two groups’ longitudinal rates of growth in achievement—that is, the 
average change in scores per year—were compared. Finally, to respond to the 
third research question (Does the number of years of City Connects exposure, or 
“dosage,” have an association with rate of growth?), the analyses compared inter¬ 
vention students’ achievement growth rates for those who spent longer and 
shorter periods in City Connects schools. The analysis employed a multilevel 
longitudinal growth model, which was designed to account for multiple years 
of achievement data over time for each student, as well as the grouping of 
students within schools. 

Results 


Descriptive Statistics 

Table 1 shows the demographic characteristics of students in the interven¬ 
tion and comparison groups. Although the comparison schools were selected 
from communities with relatively high poverty and racial diversity, they still 
had lower proportions of low-income and historically disadvantaged minority 
students than the intervention schools. Nearly one third of CCCS students 
participated in the federally subsidized lunch program, compared to 9% of 
comparison students. 

Table 1 


Student Demographic Characteristics 



CCCS 

N = 3,216 
% 

Comparison 

N = 2,403 
% 

Male* 

49 

46 

Race*** 



White 

40 

55 

Black 

18 

10 

Hispanic 

16 

8 

Asian 

7 

10 

Other or more than one race 

19 

17 

Has Individualized Education Plan (|EP)*** 

4 

1 

Eligible for free or reduced-price lunch*** 

31 

9 


*p<.05, **p<.01, ***p<.001 for Pearson chi-square statistic 
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Among CCCS students, 16% was enrolled in a City Connects school for 
one year during the study period, 20% for two years, 29% for three years, and 
36% for all four years. Among all students in the analytic sample, 44% had 
one year of Stanford 10 mathematics scores available during the study period, 
41% had two years, 10% had three years, and 6% had data for all four years 
(the distributions for the other two subject scores were similar). Based on the 
intra-class correlation coefficients for scores in each subject, 56%—63% of the 
variation in academic achievement was attributable to differences between 
students within schools, while i2%-i5% was attributable to average differ- 

Table 2 


Stanford Achievement Test Scale Scores: Descriptive Statistics for Analytic Sample 


Grade 

2nd 

N 

Mean 

SD 

Mathematics 

CCCS Como. 

904 434 

591.8 589.6 

37.4 38.4 

Reading 

CCCS Como. 

878 434 

613.5 616.3 

36.5 37.0 

Language 

CCCS Como. 

904 434 

615.6 619.3 

41.2 42.9 

3rd 

N 

1022 

471 

988 

471 

1022 

471 


Mean 

617.3 

618.8 

632.5 

637.8 

624.1 

634.6 


SD 

3 8.2 

42.4 

36.9 

37.1 

38.6 

42.1 

4th 

N 

913 

700 

883 

700 

913 

700 


Mean 

639.0 

645.8 

648.5 

659.3 

643.0 

658.1 


SD 

35.7 

35.3 

35.8 

35.0 

38.1 

39.5 

5th 

N 

819 

726 

794 

726 

819 

726 


Mean 

657.3 

667.7 

660.5 

671.4 

656.3 

671.7 


SD 

36.6 

36.4 

32.2 

32.1 

38.6 

38.3 

6th 

N 

843 

811 

828 

811 

843 

811 


Mean 

679.5 

682.4 

680.1 

688.2 

668.9 

678.3 


SD 

40.8 

38.7 

37.6 

33.3 

39.0 

34.5 

7th 

N 

355 

825 

348 

825 

355 

825 


Mean 

670.8 

696.9 

673.0 

697.4 

664.8 

694.8 


SD 

30.0 

36.5 

26.1 

28.2 

29.0 

36.7 

8th 

N 

355 

818 

342 

818 

355 

818 


Mean 

684.2 

707.0 

685.7 

704.1 

673.0 

702.7 


SD 

33.4 

35.7 

28.1 

26.9 

30.6 

37.9 


Note. CCNX = students at schools participating in City Connects. Comp. = students 
attending comparison schools 
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ences between schools, with the remainder of the variance associated with 
differences in scores over time for each student. 

Before controlling for student characteristics, comparison school students 
had higher SAT10 scale scores on average than CCCS students in every 
subject and grade, with the exception of second-grade mathematics, and that 
difference increased with grade level. Table 2 shows the descriptive statistics. 
Academic Achievement Results 

The following section presents the results of cross-sectional, longitudinal 
growth, and dosage analyses within each academic subject. Appendix A dis¬ 
plays the complete model statistics for each of the three subject areas. 

Mathematics achievement. The cross-sectional analysis of average 
mathematics achievement level in sixth grade found that scores were signifi¬ 
cantly higher for CCCS students than for their comparison school peers with 
similar demographic characteristics. The average difference between the two 
groups, controlling for demographic factors (gender, race/ethnicity, having 
an Individualized Development Plan, and eligibility for free/reduced-price 
lunch), was 16 scale score points—an effect size greater than half a standard 
deviation. Ibis difference is one third greater than the size of the achieve¬ 
ment gap (12 scale score points) associated with eligibility for free/reduced- 
price lunch estimated in this analysis. 

The longitudinal growth analysis of mathematics achievements indicated 
that CCCS students also had a significantly higher rate of growth. As the 
slopes of the growth trend lines in Figure 2 illustrate, by 6th grade, CCCS 
students gained at a higher rate than their comparison peers. Finally, the 
analysis of dosage showed that students who were enrolled in a City Connects 
school for more years were also more likely on average to experience addi¬ 
tional gains in achievement, compared to those with fewer years of CCCS 
involvement. 

Figure 2 illustrates the differences in achievement between comparison 
students and CCCS students who remained in intervention schools for four 
years, for a hypothetical group of average students 2 in grades three through 


a All demographic covariates were grand-mean centered; the figure represents students at the 
grand mean. 
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Grade 

- Comparison -- CCCS 

six, based on model estimates. 

Figure2. Growth in Stanford 10 mathematics scores (2009-2010 to 2012-2013): Model-ad- 
justed mean scores (CCCS N = 3,216; Comparison N = 2,403). For CCCS students, assumes 
the student is enrolled in a CCCS school from 3rd to 6th grade. Figure shows estimated 
mean scores. 

Students who were male, higher-income, White, or Asian had signifi¬ 
cantly higher mathematics scores by sixth grade, on average, whereas students 
who were female, Black, Hispanic, lower-income, or had an IEP had lower 
scores. A positive interaction between being lower-income and being Black 
mitigated the net effect of these two negative factors occurring together. No 
other interactions were found between student-level characteristics, and there 
were no significant interactions between treatment status and student charac¬ 
teristics, suggesting that the relationship between treatment and mathematics 
achievement did not differ across different types of students. The only de¬ 
mographic characteristic associated with the rate of growth was income level: 
lower-income students had a slower rate of growth on average compared to 
higher-income students. 

Reading achievement. For Stanford 10 reading assessment scores, CCCS 
students again exhibited higher scores on average in sixth grade than their 
peers in comparison schools. However, the size of this gap was smaller than 
for math, and the difference was not statistically significant at an alpha level 
of .05. The average reading scale scores for CCCS sixth-graders were 11 
points (39% of a standard deviation) higher than those for their peers after 
controlling for demographic factors (gender, race/ethnicity, having an Indi¬ 
vidualized Development Plan, and eligibility for free/reduced-price lunch). 
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The number of years the student spent in CCCS schools was likewise not a 
significant factor for reading growth. However, the average rate of growth 
estimated in the longitudinal analysis was significantly higher for CCCS stu¬ 
dents than for their comparison school peers by 6th grade. Figure 3 presents 



Grade 

- Comparison - CCCS 


the estimated reading growth curves. 

Figure3. Growth in Stanford 10 reading scores (2009-2010 to 2012-2013): Model-adjusted 
mean scores CCCS N = 3,182; Comparison N = 2,403. For CCCS students, assumes the 

student is enrolled in a CCCS school from 3rd to 6th grade. 

Being Black, Hispanic, low-income, or having an IEP were factors signifi¬ 
cantly associated with lower reading achievement in sixth grade. A positive 
interaction between being Black and lower-income was present for reading 
achievement, similar to the pattern seen for mathematics. In addition, boys 
and Black students exhibited a higher estimated rate of growth in reading 
than their female and White peers. 

Language achievement. Based on the cross-sectional analysis of lan¬ 
guage assessment scores, sixth-graders participating in CCCS had higher 
scores on average than their peers in comparison schools after controlling for 
demographic characteristics (15 scale score points, or half a standard devia¬ 
tion), but the difference was not statistically significant at the .05 alpha level. 
This relationship between CCCS participation and achievement was weaker 
among lower-income students compared to higher-income students, as indi¬ 
cated by a statistically significant interaction term. As the two graphs in Fig¬ 
ure 4 illustrate, lower-income CCCS students started out with slightly lower 
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average scores in third grade than their lower-income peers in comparison 
schools, but surpassed the comparison students by sixth grade. The overall 
difference in scores for this subgroup in sixth grade was about a quarter of 
a standard deviation (8 scale score points). Among higher-income students, 
CCCS participants started out at about the same level as comparison stu¬ 
dents, but pulled ahead of them by approximately half a standard deviation 
(15 scale score points) by sixth grade. Although higher and lower income 
students exhibited different patterns, the relationship between CCCS partici¬ 
pation and sixth grade results did not reach the level of statistical significance 
in either group. 

The longitudinal analysis found that the rate of growth in language 
achievement was significantly higher for students participating in CCCS 
by 6th grade. However, the maximum time spent in CCCS schools did not 
have a relationship to the rate of language achievement growth. 



Figure 4. Growth in model-adjusted mean Stanford 10 language scores (2009-2010 to 2012- 
2013) by free/reduced-price lunch eligibility. Cross-sectional differences in language scores 

in 6th grade between CCCS and Comparison students were not statistically significant. 

Students who were male, Black, Hispanic, other races, or had an IEP had 
lower language scores on average by the sixth grade. The positive interaction 
between being Black and lower-income was again present. No demographic 
characteristic was statistically associated with the rate of growth in language 
achievement. 
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A Note on Limited English Proficiency 

English proficiency is an additional student factor relevant to achieve¬ 
ment. In Boston, which has a large immigrant population, public school 
students who are classified as English language learners have lower state 
achievement test scores on average (Boston Public Schools, 2013). In this 
study, although some student records indicated whether or not the child 
was LEP and/or bilingual, 36% of students were missing this information. 
Two alternative sets of models were estimated: one in which LEP status was 
included in the model and listwise deletion performed, thereby reducing the 
analytic sample by nearly one-fourth; and one in which all cases were re¬ 
tained, and missing data for LEP (as well as other demographic characteris¬ 
tics) represented with a dummy variable. 

In the listwise deletion models, LEP status was not a statistically signifi¬ 
cant factor for achievement in any of the three subjects. In the analysis with 
missing data dummies, LEP did not have a statistically significant relation¬ 
ship to achievement in mathematics or language, but did have a significant 
negative association with reading scores. In all three subjects, the coefficient 
for missing LEP data was significant, indicating that students with miss¬ 
ing LEP status differed from those whose LEP status was available. In both 
models, the direction and magnitude of the overall relationship of treatment 
to outcome were similar to the original models without the LEP variable. 
However, in both alternative LEP models, the treatment coefficient was 
statistically significant for language and reading achievement, in contrast to 
the nonsignificant finding for these subjects in the original model. While 
the more conservative, nonsignificant results from the original models are 
reported in this article, it appears that English language learners may have a 
different experience with CCCS than other students, a finding worthy of ad¬ 
ditional study. A study of within-school implementation changes in Boston 
Public Schools found that City Connects had significant, positive effects on 
immigrant students’ math and reading achievement at the end of elemen¬ 
tary school, and narrowed the achievement gap between English Language 
Learners and English proficient students (Dearing et al., in press). 

Discussion 

These findings suggest that participation in a systematic, comprehensive 
intervention addressing out-of-school barriers to learning is associated with 
increased academic achievement in the Catholic school context. 
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In mathematics, the cross-sectional analysis found that students at City 
Connects schools had pulled ahead of comparison schools by sixth grade 
and demonstrated significantly higher average achievement scores. Al¬ 
though achievement advantages were found for CCCS students in reading 
and language as well, those differences were smaller and did not reach the 
level of statistical significance. Mathematics learning may be more sensi¬ 
tive to improvements in noncognitive attitudes and skills—such as attention, 
self-efficacy, and study habits—and therefore more responsive to support. A 
meta-analysis of impact evaluations of a widely used counseling intervention 
to help students develop cognitive, social, and self-management skills sup¬ 
ports this hypothesis; the authors found a larger overall treatment effect size 
for mathematics than for reading (Villares et al., 2012). English proficiency, a 
factor not included in the present study, might also have some bearing on the 
smaller, nonsignificant differences found for language and reading. 

CCCS students exhibited significantly higher rates of growth in achieve¬ 
ment in each of the three subject areas compared to nonparticipating peers 
with similar demographic characteristics. This finding is consistent with the 
City Connects theory of change, which posits that students will come to 
school more ready to learn when their strengths and needs are addressed, and 
teachers will be better able to tailor their instruction when they know more 
about those strengths and needs. The schools’ enhanced ability to engage 
with families may also contribute to achievement: a dedicated school site 
coordinator facilitates relationships and communication between families, 
schools, and services. Other research has documented the effective use of 
asset-based, social justice-oriented practices in urban Catholic schools to 
successfully connect with parents and other caregivers (Scanlan, 2008). 

Mathematics was the only subject in which additional advantage accrued 
with more years of City Connects exposure. For a student who remains in a 
comprehensive support intervention over time, the benefits appear to accu¬ 
mulate and reinforce the student’s achievement gains in this subject area. 

The mathematics effect sizes found in this study are comparable to results 
from other comprehensive student support interventions. For example, the 
Harlem Children’s Zone Promise Academy reported an effect size of 0.23 
standard deviations on the New York Math Exam associated with one year 
of the intervention in middle school (Dobbie 8c Fryer, 2011), and the SEED 
urban boarding middle school found a 0.30 effect size on math achievement 
measures for students after one year of participation (Curto 8c Fryer, 2011). 
The findings are also consistent with those reported for City Connects in 
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the Boston Public Schools, where students averaged 0.21 standard deviations 
higher than nonparticipants on the state mathematics achievement test as a 
cumulative effect over five years ending in seventh grade (Walsh et al., 2014). 
A City Connects study of the cost per unit increase in effect size in public 
schools found that City Connects achieved larger achievement gains for a 
lower cost than these more resource-intensive support programs (Sibley, Rac- 
zek, Dearing, 8c Walsh, 2014). A similar efficient cost structure characterizes 
the Catholic school implementation of the program. 

Addressing the Achievement Gap 

For mathematics achievement, City Connects participation was associ¬ 
ated with reductions over time in the gap between higher and lower income 
students’ scores. Low-income students who remained in City Connects 
schools for four years were estimated to achieve near parity with middle- and 
higher-income students in comparison schools by the time they reached 
sixth grade: CCCS students who were eligible for free/reduced-price lunch 
had slightly higher estimated sixth-grade scores on average than comparison 
students who were not eligible for the free lunch program. In relation to 
language scores, however, participation in City Connects interacted in un¬ 
expected ways with poverty status. CCCS participation appeared to have a 
stronger beneficial association with sixth-grade language achievement among 
students who did not qualify for free or reduced-price lunch than among 
lower-income students (although it was not a statistically significant rela¬ 
tionship for either group)— an example of the “Matthew effect,” by which 
advantages accrue to those who already have advantages. Furthermore, the 
higher concentration of poverty in Boston schools versus comparison schools 
may have aggregate effects on low-income students and their school environ¬ 
ment that slow their academic growth and make it more difficult for them to 
reap the benefits of support programs like City Connects. Growing up in a 
neighborhood with a high concentration of poverty may have negative effects 
on children’s cognitive development, through mechanisms such as normative 
parenting practices (Greenman, Bodovsi, 8c Reed, 2011), availability and use 
of enrichment activities outside school (Dearing et al., 2009), environmental 
stressors, trauma, lack of access to healthy food, and exposure to environmen¬ 
tal toxins (Nelson 8c Sheridan, 2011). 

However, pre-existing differences between CCCS and comparison 
schools should be kept in mind as context for interpreting results. Because 



134 


Journal of Catholic Education / May 2016 


a lower percentage of comparison students than intervention students were 
low-income (see Table 1), the two groups were not equivalent, meaning that 
the effects found in the analysis could be biased by differences between the 
two groups. The analyses attempted to adjust for the lack of equivalence by 
including student poverty status as a control variable in the models. 

This study analyzes associations among achievement, treatment, and other 
factors; it does not seek to establish causal relationships. Because the study 
does not employ an experimental design with randomly assigned treatment 
and control groups, there may be unmeasured factors associated with higher 
academic achievement that systematically lead students to attend a City 
Connects school rather than a comparison school. For example, parents with 
higher levels of education, who are better equipped to support their children’s 
learning, might represent greater numbers in the comparison cities than 
in Boston. Such factors could introduce bias into the results. Analysis of 
student support interventions and their effects using an experimental design 
with random assignment, or a quasi-experimental propensity score matching 
design, would contribute evidence about causal linkages between treatment 
and outcomes. Finally, additional detail and qualitative information about 
the school contexts in which the intervention takes place would help admin¬ 
istrators identify any characteristics of the school environment that facilitate 
or impede program success. In particular, differences in language achieve¬ 
ment results based on student family income warrant additional study. 

Conclusion 

Ibis article has presented findings from a study of a coordinated, sys¬ 
temic approach to addressing out-of-school needs that is easily implemented, 
cost-effective, and shows promising results in relation to increased academic 
achievement. The results add to the body of research on meeting nonacadem¬ 
ic needs as a potential strategy for closing achievement gaps between lower- 
income and middle/upper income students. It extends what is already known 
about such interventions in the public school setting to the Catholic educa¬ 
tion context. The intervention is particularly well suited to Catholic schools, 
with their explicit commitment to educating the whole child. Given the 
lagging academic performance of urban, lower-income students in Catholic 
(as well as public) schools nationwide, it is incumbent on schools to consider 
how to mitigate the effects of poverty on their students’ academic progress. 
Free of some of the regulations that constrain operations in the public sector, 
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Catholic schools have the flexibility to try out innovative approaches in ways 
that public school systems cannot. To that end, many Catholic schools use 
their mandate to educate the “whole child” as an opportunity to promote a 
safe, positive school culture and collaborative community support structures 
that help students thrive (Goldschmidt 8c Walsh, 2012). If urban Catholic 
schools are to achieve their vision of holistic education of the whole child, it 
will be critical to offer families living in poverty coordinated, systematic help 
to address the multiple barriers they face. 
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Appendix A 


Table A1 

Longitudinal Growth Models of Stanford 10 Achievement Scores (2009-2010 to 2012-2013): 


Fixed Effects 



Mathematics 

Reading 

Language 


Coeff. (SE ) 

Coeff. 

(SE1 

Coeff. 

(SE1 

Intercept 

667.35 *** 4.98 

673.15 *** 

4.79 

663.89 *** 

5.92 

Any CCCS Treatment 

16.26 ** 5.80 

10.77 

7.00 

14.68 I 

8.37 

Male 

4.56 ** a 1.26 

-1.58 

1.15 

-8 54 *** a 

1.25 

Black 

- 11.10 *** 2.06 

-11.65*** 

1.75 

-11.51 *** 

2.13 

Hispanic 

-6.29 * 2.79 

-6.85 * 

2.77 

-7.46 ** 

2.44 

Asian 

14.99 *** a 3.68 

-0.57 

3.21 

2.27 

2.58 

Other/more than one race 

0.64 2.00 

-3.19 

2.09 

-3.39 * 

1.66 

IEP 

-24.89 *** 3.07 

-21.90 *** 

3.75 

-23.99 *** 

3.21 

Free/reduced-price lunch 

-12.15 *** a 1.88 

-9.66 *** 

2.02 

-4.22 I 

2.50 

Any CCCS x Lunch 




- 6.86 ** 

2.55 

Lunch x Black 

10.15*** 2.55 

10.77 ** 

3.14 

10.23 ** 

2.99 

TIME slope: 

Intercept 

15.01 *** a 0.75 

11 78 *** a 

0.35 

11.61 *** 

0.59 

Male 

0.06 0.33 

0.85 * 

0.37 

0.54 

0.34 

Black 

0.92 0.73 

1.75 ** 

0.55 

1.00 

0.67 

Hispanic 

-0.15 0.80 

0.95 

0.60 

0.65 

0.70 

Asian 

0.17 1.10 

-0.47 

0.74 

-0.32 

1.23 

Other/more than one race 

0.35 0.54 

0.31 

0.44 

0.48 

0.55 

IEP 

0.23 0.95 

1.81 1 

1.06 

1.60 

1.14 

Free/reduced-price lunch 

-1.05* 0.52 

-0.06 

0.55 

0.09 

0.54 

TIME-squared slope: 
Intercept 

-3.04*** 0.16 

-1.03 *** 

0.09 

-0.45 ** 

0.15 

Dosage slope: 

Intercept 

4.64*** 1.00 

2.48 ** 

0.95 

3 74 *** 

0.74 

Maximum CCCS dosage 

0.77* 0.31 

-0.19 

0.26 

0.23 

0.27 


I p<.io; *p<.o 5 ; **p<.oi; ***p<.ooi; 
a Slope allowed to vary randomly across schools 
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Table A2 


Longitudinal Growth Models of Stanford 10 Achievement Scores (2009-2010 to 
2012-2013): Random Effects 



Mathematics 

Reading 

Language 

Variance Components 




Levels 1 and 2: 




Within-student variance 

277.10 

217.55 

409.34 

Between-student variance 

823.20 *** 

758.52 *** 

827.79 *** 

TIME slope variance 

22.44 *** 

1.22 ** 


Male slope variance 

15.90 ** 


14.35 ** 

Asian slope variance 

152.11 *** 



Free lunch slope variance 

35.03 * 



Level 3: 




Between-school variance 

177.81 *** 

98.76 *** 

177.33 *** 

Reliability Statistics 




Level-1 intercept 

0.48 

0.65 

0.76 

TIME slope 

0.10 

0.08 


Level-2 intercept 

0.94 

0.95 

0.97 

Male slope 

0.43 


0.39 

Asian slope 

0.56 



Free lunch slope 

0.40 



Variance Explained 




Between students 

34% 

54% 

31% 

Between schools 

27% 

70% 

44% 

Total 

35% 

61% 

35% 


*p<- 05 ; **p<. 01 ; ***p <.001 



